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Technical field 

This invention relates to the multimedia information classification arranging 
apparatus which can perform efficiently and flexibly classification and arrangement of 
the multimedia information group intermingled as various media information, such as 
not only text information but picture information, speech information, etc. WWW of 
the Internet (World Wide Web.) It is hereafter written as WWW, and also a 
classification and arrangement of an accessible multimedia information group are also 
possible. 

Background art 

Now, the multimedia data of not only text information but picture information, speech 
information, etc. is stored in large quantities, and it is accessible. The increase 
especially of the information which is on WWW (World Wide Web) with 
development of the Internet is being enhanced. 

The various multimedia data of text information, picture information, speech 
information, etc. is stored in large quantities, and it is accessible. 
As a result, it is becoming very difficult to retrieve required information exactly from 
sources of information. 

The following are known as a classification and layout process art of the information 
in a conventional information retrieval system and the search system concerned. 
The text key search system is known by the 1 st. Assign suitably 1 or two or more text 
information which express the contents to the multimedia data used as a retrieval 
object, and in search, Texts, such as a word, are specified as a search key and the 
method of detecting the multimedia data in which the same text information as the 
search key concerned is assigned is known. When the media information of a retrieval 
object is text information, the contents of text information will be searched by a text 
key, and the text key search concerned can be performed in general appropriately in 
many cases. What sorts and displays in a list the data hit by search as an example of a 
classification / arrangement of the information in this text key search system is 
mentioned. 



The text full text retrieval system is known by the 2nd. This is an especially effective 
search system, when the multimedia data used as a retrieval object is text information. 
Although some methods are proposed, for example, all the words showing the feature 
of the contents are automatically extracted from the text whole sentence as 
pretreatment, and the search key file is generated and assigned. In search, texts, such 
as a word, are specified as a search key and the text information to which the same 
text as the search key concerned exists in a search key file is detected. Like the search 
method which used the 1st text of the above as the key also by this method, if a 
suitable text key is not given for every picture information, it is difficult about media 
information other than text information, for example, picture information, to refer to 
the search which used the text as the key appropriately. 
What sorts and displays in a list the data hit by search is mentioned like [ the example 
of a classification / arrangement of the information in this text full text retrieval 
system ] a text key search system. 

The image retrieval art by the image pattern collation which specified the particular 
part of the picture and used the part image as the search key is known as art of 
searching picture information by the 3rd. According to this image retrieval method, it 
is possible to search picture information with the specified part image. 
What sorts and displays in a list the data which also hit the example of a classification 
/ arrangement of the information in the image retrieval system by this image pattern 
collation by search is mentioned. 

The sound search technique by sound pattern matching is studied as art of searching 
sound information by the 4th. When a user specifies the sound data (humming etc.) 
used as a key as a sound information search system, a system searches the audible 
signal and similar sound data, and what shows a user information, including a track 
name etc., is mentioned. 

There is art of searching video information in the 5th. As the method of image 
retrieval, segmentation of the image is carried out beforehand, The contents of each 
segment are recognized using art, such as an operation recognition algorithm, speech 
recognition, telop recognition, and the method of performing label attachment in 
alignment with information content, and the method of creating and searching a 



search key by sketching the objective move direction are studied. The system with 
which specific operation of search of the highlight scene produced as this video 
information search system in play-by-play broadcasting of baseball, such as a home 
run, a shot, the goal, soccer, etc., the smash in tennis, a volley, etc. is searched is 
being developed. 

However, as for the conventional information retrieval system, the effective high 
search method of practicality does not exist about any media information other than 
text information, for example, picture information or speech information. 
In the 1 st conventional text key search system, it is difficult to carry out appropriately 
about picture information or speech information. That is, since it is difficult to express 
the feature of a picture in a text exactly and flexibly even if it assigns two or more 
keywords to one picture information, according to the intention which a retrieving 
person searches, execution of suitable and flexible image retrieval is difficult by a 
keyword. Many pictures were collected, and although it was possible to have 
displayed them in order, when the number of pictures increased, there was a problem 
that it became difficult to discover the target picture from the inside. 
It is difficult to carry out appropriately about picture information or speech 
information in the case of the conventional whole sentence text browsing system [ 
2nd ] as well as the 1 st text key search system. That is, neither picture information nor 
speech information has text information from the first, and extraction of text 
information cannot be expected. 

According to the image retrieval system by the 3rd conventional image pattern 
collation, it is possible to search picture information with the specified part image. 
However, the retrieving person needs to prepare and specify the part image which a 
picture to search has. A picture [ want / a retrieving person ] may not be clear with 
what kind of part image, or the part image used for search cannot always prepare, and 
search becomes difficult. The picture which the retrieving person itself wants to 
search may not be grasped clearly. That is, there is a case of a picture to search where 
specification of the picture itself cannot be performed even if partial although the 
contents of **** can be specified. There is [ whether there is any picture which 
something can use vaguely, and ] a retrieving person to search by trial and error. In 
such a case, pliability is missing and just matching of a part image is insufficient. 



In the sound search system by the 4th conventional sound pattern matching, although 
a general user's humming is one note, since the phenomenon of a pitch being [ more 
than seminote ] unsteady, or making a mistake in a pitch happens, suitable search is 
difficult. 

Also in the 5th conventional video information search system, it is exactly difficult for 
image contents to carry out label attachment or to describe movement of the subject in 
an image, and the actual condition is that the video information search method with 
flexibility applicable to a general image does not yet exist. 
As mentioned above, there is a problem of "difficulty of the suitable key designation 
for narrowing down information" in the conventional information retrieval technique 
in common. As much information as possible is shown to a retrieving person as a 
means to solve this problem, and there is approach of getting a retrieving person to 
look for desired information out of it. When the retrieval object comprises 
information units (the picture itself, a picture, a text mixture document, etc.) including 
a picture, since it excels in list nature for human being, a user becomes possible [ 
discovering desired information comparatively easily out of a lot of shown 
information ]. However, the following problems arise. 
As for the 1st problem, the retrieval device is operating on a server. 
It is the problem that a lot of pictures will have to be transmitted when displaying a 
picture on a client via a network, and a retrieving person's waiting time will become 
long. 

Like a picture and a text mixture document, the 2nd problem is a problem that the 
picture is not necessarily expressing the contents of the information unit exactly, 
when information other than a picture is also included in an information unit. That is, 
the picture which an information unit includes is used in many cases only for the 
ornament etc., and, in such a case, a retrieving person cannot find a desired 
information set exactly by the inspection search based on the picture concerned. 
Since the 3rd problem lacks in list nature even if it presents information as it is when 
the retrieval object comprises an information unit (only the document data of only a 
text, and sound data only picture image data) which does not include a picture, a 
user's retrieval effectiveness does not increase. 



An object of this invention is to provide the multimedia information classification 
arranging apparatus and classification configuration method which reduced the data 
volume to transmit and secured the list nature of information content to a retrieving 
person, and grasp ease, when performing an information display on a client via a 
network in view of the above-mentioned problem. 

In this invention, the various characteristic quantity in which multimedia information 
has multimedia information, such as text information, picture information, and speech 
information, is used efficiently. 

Therefore, it searches efficiently and flexibly and search results are aimed at visual 
and providing the multimedia information classification arranging apparatus and 
classification configuration method which can carry out classification and 
arrangement and can be displayed so that intelligibly. 

The indication of an invention 

To achieve the above objects, the multimedia information classification arranging 
apparatus of this invention, What carried out grouping of the congener and different- 
species media information which relate to the same object from the set of media 
information including picture information, text information, and speech information is 
used as an "information set", and the information set is used as a batch. And the icon 
which displays the contents visually is assigned to each media information which 
constitutes an information set, and the "information set icon" which compounds these 
icons and expresses the contents of said information set visually is used. 
The information retrieval technique of conventional technology differs between the 
point of using an "information set" as a unit of Data Processing Division in this 
invention, and the point using an "information set icon", clearly. The "information 
set" of an application concerned compounds each media information itself like the 
multimedia information of conventional technology, and it is not unifying, and it 
relates with the common concept and the multimedia information of the conventional 
one compositive type differs clearly. The Data Processing Division unit of an 
"information set icon" is an information set. 



It is the icon which assigned the icon applicable to each media information which 
constitutes the information set, such as a text, a picture, and a sound, compounded 
these icons, and was made into the form corresponding to the information set as a 
Data Processing Division unit. 

While according to this information set icon it can create so that compression 
efficiency may be high, and there may be little data transfer quantity and may end, 
What kind of media information is included in the information set can glance, it can 
grasp easily, and the list nature of information content to a retrieving person and grasp 
ease can be made to secure. 

The multimedia information classification arranging apparatus of this invention, The 
information set acquisition part which acquires media information by making an 
information set into a unit, The icon which displays the contents visually is assigned 
to each media information which constitutes said information set, The information 
visualization part which generates the information set icon which compounds these 
icons and expresses the contents of said information set visually, The feature quantity 
extracting part which extracts the ingredient of characteristic quantity from each 
media information of each information set, The axiation part which sets up the 
classification configuration space which assigned the attribute chosen from the 
attribute of the characteristic quantity extracted from each media information included 
in each acquired information set as an axis of the space which arranges an information 
set icon, and was provided with one or more axes, Based on the attribute and the 
ingredient of characteristic quantity of the characteristic quantity of the media 
information which each information set has, it had a classification placement part 
which carries out classification arrangement of the information set icon, and 
information display sections which display said classification configuration space and 
said information set icon in said classification configuration space. 
By generating and using the information set icon which compression efficiency is 
high, and there is little data transfer quantity as data used for presenting of 
information, and ends by the above-mentioned composition, The data volume to 
transmit can be reduced, and the waiting time to a retrieving person's information 
display can be shortened, and the list nature of information content to a retrieving 



person and grasp ease can be secured. A user's retrieval effectiveness can be raised by 
arranging to the neighborhood the information set icon related in an information set 
icon by carrying out classification arrangement in a feature space according to the 
characteristic quantity, or arranging a picture with the specific feature at a specific 
place. 

Processing by the above-mentioned multimedia information classification arranging 
apparatus is divided into two phases, an information set icon generation phase and the 
information retrieval phase using an information set, An information set icon 
generation phase can be beforehand performed as pretreatment, the information set 
icon of the information set used for the search in an information retrieval phase can be 
summarized, and generation and preparation of can also be done. In [ according to this 
composition ] an information set search phase, It can only accelerate processing that it 
is not necessary to generate an information set icon each time from the information set 
obtained as a result of search, and classification arrangement of the information set 
icon corresponding to the searched information set may be carried out in classification 
configuration space. 

Next, other composition of the multimedia information classification arranging 
apparatus of this invention, As mentioned above, compound the icon corresponding to 
each media information which constitutes an information set, and an information set 
icon is not created, According to the characteristic quantity, classification 
arrangement of the icon is carried out in classification configuration space by making 
the icon corresponding to each media information into a bundle, and it displays on 
said locating position, changing each icon in time according to a display order in the 
locating position concerned. 

That is, the information set acquisition part which acquires media information by 
making said information set into a unit, The information visualization part which 
assigns the icon which displays the contents visually to each media information which 
constitutes an information set by making said information set into a unit, The feature 
quantity extracting part which extracts the ingredient of characteristic quantity from 
each media information of each information set, The axiation part which sets up the 
classification configuration space which assigned the attribute chosen from the 



attribute of the characteristic quantity extracted from each media information included 
in each acquired information set as an axis of the space which arranges an icon, and 
was provided with one or more axes, The classification placement part which carries 
out classification arrangement of the icon to each media information included in said 
information set in said classification configuration space based on the attribute and the 
ingredient of characteristic quantity of the characteristic quantity of the media 
information which each information set has, It had information display sections 
displayed on said locating position, having decided the display order of the icon to 
each media information which is information display sections which display said 
classification configuration space and said icon, and is included in said information 
set, and changing these icons in time. 

By generating and using the icon which compression efficiency is high, and there is 
little data transfer quantity as data used for presenting of information, and ends by the 
above-mentioned composition, Since it displays being able to reduce the data volume 
to transmit, and being able to shorten the waiting time to a retrieving person's 
information display, and changing each icon according to a display order, the list 
nature of information content to a retrieving person and grasp ease are securable. 
It divides into two phases, an information set icon generation phase and the 
information retrieval phase using an information set, also in the above-mentioned 
composition, An information set icon generation phase can be beforehand performed 
as pretreatment, the information set icon of the information set used for the search in 
an information retrieval phase can be summarized, and generation and preparation of 
can be done. 

When two or more ingredients of the characteristic quantity which the feature quantity 
extracting part extracted exist about media information with a certain information set 
in the above-mentioned composition, a classification placement part, If classification 
arrangement of each icon changed and displayed on two or more positions according 
to said information set icon or a display order based on two or more ingredients of 
each of characteristic quantity is carried out, even if two or more ingredients exist in 
characteristic quantity, classification arrangement can be appropriately carried out in a 
feature space. 



In assignment of the icon according [ on the above-mentioned composition and ] to an 
information visualization part, If either of the icons currently prepared beforehand is 
chosen and the display information of said selected icon is embellished according to 
the contents of corresponding media information, the grasp which reflected the 
contents of information for the icon and information set icon for every media — it is 
easy and can be considered as the suitable thing for search. 
Next, in the above-mentioned multimedia information classification arranging 
apparatus said axiation part, It has an axial resetting function which resets assignment 
of the attribute of the characteristic quantity to each axis of a classification 
configuration space axis, and resets the classification configuration space provided 
with one or more axes, Said feature quantity extracting part extracts the ingredient of 
characteristic quantity according to the axial re set by said axiation part, It is preferred 
that said classification placement part carries out classification arrangement of each 
information set icon into said classification configuration space according to the 
ingredient of said extracted characteristic quantity, and said information display 
sections display the media information according to the viewpoint over said reset 
classification configuration space. 

The search results which the retrieving person performed can be seen by the above- 
mentioned composition, re retrieval is performed interactively, information can be 
narrowed down flexibly or search can be continued one after another by a further 
different cut end using the reset classification configuration space axis. That is, search 
results are seen, by trial and error, another characteristic quantity can be specified, the 
axial re set of classification configuration space can be performed, an information set 
icon can be rearranged using the classification configuration space using the reset 
axis, and redisplay can be performed. 

Next, in the above-mentioned multimedia information classification arranging 
apparatus said axiation part, The classification configuration space which resets 
assignment of the attribute of the characteristic quantity to classification configuration 
space each axis, and has already been displayed is reset, Said classification placement 
part carries out reclassification arrangement of each information set icon to the reset 
classification configuration space, When said information display sections display 



signs that reclassification arrangement of each information set icon is carried out, it is 
preferred to make it move by predetermined unit and to display the media information 
currently displayed from the position currently displayed before reclassification 
arrangement to the position displayed after reclassification arrangement. 
When according to the above-mentioned composition classification configuration 
space is reset and reclassification of the information set icon is carried out, the 
situation how the locating position of each information set icon changed can be 
checked visually, and the convenience of search of an information set and sorting 
processing improves. 

The function which said classification placement part fixes to the specific position on 
the classification configuration space where the user specified the information set icon 
which the user selected in the above-mentioned multimedia information classification 
arranging apparatus, When carrying out reclassification arrangement of each 
information set icon to the classification configuration space where the axis was reset, 
it is preferred to have a function which fixes to said specific position the information 
set icon which said user selected, and carries out reclassification arrangement only of 
other information set icons according to classification configuration space. 
The information set icon currently easily observed since the display position of the 
information set icon to observe is fixed to a specific position according to the above- 
mentioned composition can be found, Since an information set icon similar in the 
characteristic quantity by which axiation was carried out to the information set icon 
currently observed is arranged near, grasp of the relation between information sets 
becomes easy. After taking notice of a specific information set, a trial-and-error 
reclassification display can be performed. 

Next, the multimedia information classification arranging apparatus of this invention, 
The multimedia information classification arranging apparatus of this invention is 
realizable using a computer by recording and providing with the processing program 
of the multimedia information classification arranging apparatus of this invention the 
recording medium in which computer reading is possible. 



The best gestalt for inventing 



Hereafter, the embodiment of the multimedia information classification arranging 
apparatus of this invention and a classification configuration method is described, 
referring to Drawings. 

(Embodiment 1) 
The multimedia information classification arranging apparatus of Embodiment 1 of 
this invention is explained. The multimedia information classification arranging 
apparatus of this Embodiment 1 uses as an "information set" what carried out 
grouping of the congener and different-species media information which relate to the 
same object from the set of media information including picture information, text 
information, and speech information, and the information set is used for it as a batch. 
And the icon which displays the contents visually is assigned to each media 
information which constitutes an information set, and the "information set icon" 
which compounds these icons and expresses the contents of said information set 
visually is used. 

First, the concept of the information set used as the batch of the information used in 
the multimedia information classification arranging apparatus of this invention is 
explained. Next, the example of an equipment configuration of the multimedia 
information classification arranging apparatus of this Embodiment 1 is shown, and the 
flow of processing of the multimedia information classification arranging apparatus of 
this Embodiment 1 is shown, referring to a flow chart. 

An information set is used for the multimedia information classification arranging 
apparatus of this invention as a batch of information. It is the batch which carried out 
grouping of the congener and different-species media information which are related 
with this information set to the same object from the set of media information 
including picture information, text information, and speech information. Drawing 1 is 
a figure showing briefly the concept of the "information set" of this invention, two or 
more congener or media information of a different kind which had relevance in the 
one information set 1 mutually as shown in drawing 1 — a bundle — correlation 
******. the example of the information set 1 ~ four congener, the picture information 
la, the picture information lb, the speech information lc, and 1 d of keyword 



information, or media information of a different kind — a bundle — correlation 
******. These four congener or media information of a different kind are collected 
with the concept of information with the relevance of "the personal computer of F 
company", for example. 

Signs that the information set of this invention is generated from the multimedia 
information in conventional technology, etc. are also shown in drawing 1 . 500 is the 
multimedia information in conventional technology, and two or more media 
information is united with one, and it is compounded. For example, it is HTML file 
data etc. The information set acquisition part 1 0 is for acquiring the information set of 
an application concerned, for example, can incorporate the multimedia information of 
a conventional type, and can generate an information set. 

The procedure of information set generation is as follows. The information set 
acquisition part 10 extracts and collects image regions, and collects other media 
information, such as text information relevant to the picture concerned, other picture 
information and moving image information, and speech information, further. For 
example, an HTML file will be analyzed and acquired if it is text information. If an 
example of an analytic method is given, the text in a picture and a near position will 
assume that it is connected with the picture in contents, and will make the number of 
line feed between both one rule of thumb as a picture and a measure of the position 
nearness of a text. It is possible to guess the degree of association of a picture and a 
text from HTML tag structure. If many tags exist, it can surmise that both degree of 
association is low, and when <P>, <H1>, <HR>, etc. which are the tags showing 
especially the break of a text exist, it can be surmised that a degree of association is 
low. Since relevance with a picture is deep, the ALT attribute showing the text used 
as substitution of the picture given to <IMG> which is a tag for displaying a picture 
can assign a large degree of association. Thus, let the kind and number of tags which 
are contained between a picture and a text be one rule of thumb. Thus, the information 
set acquisition part 10 collects the media information relevant to a picture. The 
collected media information is related with a bundle as media information in relation 
to the same object, and the information set acquisition part 10 uses it as an 
information set. In the example of drawing 1 , the picture information la was 
extracted, the text information lb relevant to the picture information la, the speech 



information lc, and 1 d of keyword information were associated with the common 
concept, and the information set 1 was generated. In the conventional multimedia 
information 500, these picture information la, the text information lb, the speech 
information lc, and 1 d of keyword information are not necessarily associated with 
the common concept, and each media information is scatteringly managed on 
management. 

Since the information set was related with the bundle as media information which 
each media were not compounded and relates to the same object, it can add and delete 
media information flexibly. When adding new media information to the multimedia 
information in this conventional technology, it will embed in search of the portion 
which embeds the media information added to multimedia information, the processing 
to compound will be needed, and it will be necessary to remake the multimedia 
information itself. When the voice data about the new media information added on the 
other hand when using the "information set" of an application concerned, for example, 
"the personal computer of F company", is obtained, what is necessary will be just to 
increase correlation of the voice data concerned to the information set 1 simply. The 
information set of an application concerned compounds each media information itself, 
and is not unifying, it is only relating with a common concept, processing special to 
the contents of media information themselves is not performed, and the multimedia 
information of the conventional one compositive type differs clearly. 
An information set is described. From the HTML document in which the picture and 
text on WWW were intermingled [ in / as an information set / correlation ] in picture 
information and text information, for example, an image region is extracted as picture 
information, and the text of the circumference of the picture of an HTML document is 
extracted as text information, and both are associated. As other examples, the word 
considered to be important for example, based on the tf-idf method is extracted as 
keyword information out of text information. If it is correlation of the picture 
information, speech information, and text information of video, For example, a 
dynamic image part is extracted as dynamic image information, and voice data is 
extracted from the XML file containing video and the movie data containing a sound 
as speech information, the text of the circumference where movie data is embedded is 
extracted as text information, and each is associated. It is good also considering what 
followed the link from the HTML data in which the link is stretched to other data, and 



extracted and generated the picture information of the link destination concerned, text 
information, speech information, etc. as an information set. Two or more text 
information is associated not only to that in which picture information and text 
information exist with the gestalt of one file from the beginning like an HTML 
document but to one picture information, or also when [ that ] two or more picture 
information is conversely associated to one text, it may be. URL (Universal Resource 
Locator) can be included as some information sets. By including URL in some 
information sets, the information set by which classification arrangement was carried 
out is chosen so that it may mention later, and it becomes possible to display the page 
of the web which contains the information set concerned based on the URL. 
Without carrying out the editing process of each media information itself which 
belongs as one of the features of this information set, the picture information can 
mention the point that the various characteristic quantity which each media 
information has can be processed as characteristic quantity of the whole information 
set, while speech information exists with speech information with picture information. 
When carrying out classification arrangement based on characteristic quantity by 
making an information set into a batch so that it may mention later, the various 
characteristic quantity which each media information belonging to an information set 
has is processed as characteristic quantity of the whole information set, and the 
position by which classification arrangement of the information set icon is carried out 
is decided. 

The multimedia processing of conventional technology differs in processing using the 
information set of this this invention as a batch clearly. The multimedia processing of 
conventional technology mainly points out the following two processings. The 1st 
meaning of the multimedia processing of conventional technology means that one 
device is what can treat two or more media called picture information, text 
information, and speech information. In this case, although two or more media can be 
treated with one device, the batch itself is every media information and each media 
information, such as picture information, text information, and speech information, is 
processed individually. The 2nd meaning of the multimedia processing of 
conventional technology means that the batch itself is the multimedia data in which 
the media which are plurality were unified. In this case, it is in the state where picture 
information and speech information were embedded into text information, or the link 



acquired. This information set 3 shall be the Data Processing Division unit associated 
with the common concept of "Mozart" , and three media information about a musician 
"Mozart" shall be included. Here, the sound data 3a, the image data 3b, and the text 
data 3c are contained. It is not unified, or these each media information is not 
compounded, but is [ only being related as the information set 3, and ]. 
Next, the characteristic quantity of each media information is extracted. Suppose that 
the characteristic quantity "a musical section, a minor key, and a piano" was obtained 
as a feature of the sound data 3 a. Suppose that it was considered as the feature of the 
image data 3b, and the characteristic quantity "background:black and an object rperson 
face (field flesh color)" was obtained. Suppose that the characteristic quantity "Mozart 
and the piano concerto No. 20th" was obtained as a feature of the text data 3c. The 
technique of extracting characteristic quantity from each media information shall be 
mentioned later. 

Next, the icon according to the characteristic quantity is assigned to each media 
information. The assigned icon is embellished according to the characteristic quantity. 
In this example, a piano keyboard-like icon is assigned to the sound data 3 a. In order 
to express that it is a minor key, the color of a piano keyboard is changed blue, for 
example. The icon which started to the image data 3b centering on the complexion 
range which shows a person face, and performed image data compression, such as 
DCT, is assigned. The text icon which consists of the text sentence "Mozart and the 
piano concerto No. 20th" is assigned to the text data 3c. 

Next, the icon assigned to each media information is compounded, and one 
information set icon is generated. In this example, the information set icon 3d was 
obtained to the information set 3. 

When the data of a part of media information is missing among the media information 
which constitutes an information set, it is also possible to compensate chipped data 
and to create an information set icon. Since the inspection nature in the case of an 
information classification display is affected when especially image data is missing, it 
is preferred to compensate image data. For example, the feature of sound data in a 
certain information set only "musical section, It is a minor key and a guitar", the 
contents features of text data are "the piano performance house A and the allegro 



was stretched for example, and the data itself had carried out integrated processing. 
On the other hand, the information set which related with the bundle the congener and 
different-species media information which had relevance mutually among media 
information including picture information, text information, and speech information is 
used for the information set of this invention as a batch. 
Each media information differs from what does not relate mutually and is processed 
individually like the multimedia information of the 1st conventional meaning of the 
above. 

In this invention, although it becomes the set with which the data of two or more 
media information was associated, integrated processing of the data of two or more 
media itself was not carried out like the multimedia information of the 2nd 
conventional meaning of the above, and each media information is maintained in the 
state where it was collected and acquired. If new media information is given, it will 
become possible to make it belong to the information set which is relevant among the 
existing information sets simply, to generate a new information set, when there is no 
information set of relevant existing, etc. If integrated processing of the data of two or 
more media itself is carried out like the multimedia information of the 2nd 
conventional meaning of the above, the addition of new media information cannot be 
performed easily. 

Next, the "information set icon" used in this invention is explained. The information 
retrieval technique of conventional technology differs also in the point using an 
"information set icon" clearly. The Data Processing Division unit of an information 
set icon is an information set. 

An icon applicable to each media information which constitutes the information set, 
such as a text, a picture, and a sound, is assigned, these icons are compounded, and it 
is an icon of the form corresponding to the information set as a Data Processing 
Division unit. 

Drawing 2 is a figure showing the example of an information set icon, and an example 
of the generation procedure of the information set icon concerned. 
First, an information set is acquired. Now, suppose that the information set 3 was 



moderato No. 13th D minor", and when there is no image data, an information set 
icon is created according to an icon creation procedure which is described below. 
First, an artist dictionary is referred to for "the piano performance house A", and a 
mug shot is acquired. Next, the background of a mug shot is made into yellow (it 
corresponds to D minor). Next, the picture of a guitar is stuck on a background. Next, 
an information set icon is created, attaching the label "the piano performance house A 
and the allegro moderato No. 13th D minor", and complementing the image data 
which was missing. 

As mentioned above, while according to the information set icon compression 
efficiency is high, and there is little data transfer quantity and it ends, what kind of 
media information is included in the information set can glance, it can grasp easily, 
and the list nature of information content to a retrieving person and grasp ease can be 
made to secure. 

Next, the example of composition of the multimedia information classification 
arranging apparatus of this Embodiment 1 is shown in drawing 3 . As shown in 
drawing 3 , the multimedia information classification arranging apparatus is provided 
with the information set acquisition part 10, the feature quantity extracting part 20, the 
information visualization part 30, the axiation part 40, the classification placement 
part 50, and the information display sections 60. 70 is accessible network networks, 
such as the Internet, and the source 80 of multimedia information is in an accessible 
state via the network 70. It is assumed that the information set which is a set of the 
media information where various multimedia information, such as picture 
information, text information, and speech information, is stored in the source 80 of 
multimedia information, and which has relation mutually in this example is also 
stored. 

In the example of an equipment configuration of drawing 3 , the information set 
acquisition part 10 has composition provided with the information set reading part 11, 
the information set collecting part 12, the information set collection key input section 
13, and the storage 13. The information set collecting part 12 is a portion which 
collects information sets. 



The information sets which exist in the source 80 of multimedia information via the 
network 70 are collected. 

The information set reading part 1 1 can also receive the input of an information set 
directly from storages, such as CD-ROM, etc. Thus, when the information set 
acquisition part 10 has composition suitably provided with both the information set 
reading part 11, and information set both [ either or ] 12, The function to receive the 
input of an information set can be directly held selectively from storages, such as a 
function, CD-ROM, etc. which collect the information sets which exist in the source 
80 of multimedia information via the network 70, etc. 

The example of composition of the information set collecting part 12 explained by 
this Embodiment 1 is provided with the storages 13, such as a hard disk, and the 
information set collection key input section 14. It communicates with the source 80 of 
multimedia information on the network 70 via a communication interface. The storage 
13 can be used when it stores the collected information sets. The information set 
collection key input section 14 is a portion which specifies the range collected using a 
keyword in collection of an information set. If information sets are collected without 
specifying anything when there is an information set stored in the source 80 of 
multimedia information on a network in large quantities, the data volume to collect 
will become very large. So, in order to narrow down the range beforehand in 
collection of an information set, when a keyword is inputted from the information set 
collection key input section 14, the information set collecting part 12 collects 
information sets with the keyword concerned. 

The feature quantity extracting part 20 is a portion which extracts characteristic 
quantity from each media information of an information set. Although not illustrated 
to drawing 3 , the feature quantity extracting part 20 is equipped with the function for 
extracting various characteristic quantity from each media information. 
For example, to picture information, DCT transformation coefficient characteristic 
quantity, wavelet conversion factor characteristic quantity, HSI color histogram 
characteristic quantity, etc. are mentioned. By setting DCT transformation coefficient 
characteristic quantity as a classification configuration space axis, the classification of 



the information by the feature of the spatial frequency component of picture 
information and arrangement are attained. By setting wavelet conversion factor 
characteristic quantity as a classification configuration space axis, the classification of 
the information by the feature of the low frequency portion of spatial frequency, i.e., 
feature of the outside of the rereeling reel of the object especially reflected in a 
picture, of picture information and arrangement are attained. Although wavelet 
transforms are also a waveform and frequency conversion as well as DCT 
transformation, it can change with position (time) information maintained. By setting 
HSI color histogram characteristic quantity as a classification configuration space 
axis, the classification of the information based on the sexual desire news of a picture 
and arrangement are attained. According to the HSI color histogram, the feature of 
how many human being's skin fields are included and a picture can be held well. 
For example, to text information, the characteristic quantity which shows existence of 
a specific word, the characteristic quantity of the use count of a specific word, etc. can 
be mentioned. The word set considered to be effective in a text being characterized 
beforehand is chosen, The relative importance of each word is measured using the tf- 
idf method (term frequency-inverse document frequency), and the feature vector 
which has a tf-idf value of a word as an element is extracted from each text by 
enumerating them. 

For example, to speech information, voice frequency characteristic quantity, audio 
amplitude characteristic quantity, audio time change characteristic quantity, etc. using 
short time frequency analysis (Fourier transform and MFCC) can be mentioned. The 
feature vector obtained as a result of MFCC analysis or LPC cepstrum analysis is 
used, and characteristic quantity extraction is attained according to the feature of 
voice frequency, i.e., audio height, tone quality, etc. According to voice frequency, 
the exception of the cry of an animal, male voice, and female voice can express [ 
exception of a sounding body ] the feature of speech information, such as a person of 
voice, if accuracy goes up. Characteristic quantity extraction of the speech 
information according to audio size is attained with audio amplitude characteristic 
quantity and time change characteristic quantity. 

As for the feature quantity extracting part 20, it is preferred for the addition of a 
function which extracts characteristic quantity from each media information, 



updating, deletion, etc. to be possible. If provided by DSP (digital signal processor) 
etc. by making a characteristic quantity extract function into a micro program, if 
needed, the contents are rewritten easily and can be added. 
As opposed to each media information which constitutes each information set in 
which the information visualization part 30 is stored in the source 80 of multimedia 
information, It is a portion which generates the information set icon which assigns the 
icon which displays the contents visually, compounds these icons, and expresses the 
contents of the information set visually. As for an information set icon, it is preferred 
to generate, before the information classification layout process by a user is 
performed. 

The generation processing procedure of the information set icon by the information 
visualization part 30 is as having explained above using drawing 2 . In the procedure 
explained by drawing 2 „ two or more techniques are among processings which assign 
an icon according to the characteristic quantity (contents) of media information. 
A "contents-icon translation table" is used for the 1st. This prepares the translation 
table which described the contents of each media information, and correspondence of 
the icon assigned to it. If the characteristic quantity (contents) of each media 
information is described by the contents-icon translation table, a corresponding icon 
can be assigned easily. 

It is the method of using for the 2nd "the rule changed into an icon from the contents 
feature" beforehand described with the help. A program including the rule changed 
into an icon from the contents feature is provided. 

"Pair of contents feature-icon" data is prepared for the 3rd in large quantities, the map 
changed into an icon from the contents feature is automatically learned in it, and there 
is a method of using it in it. The expert system which has as knowledge a rule 
changed into an icon from the contents feature is provided. 

When the contents of each media information cannot finish being expressed with the 
above-mentioned processing, it changes into two or more icons, or the difference 
information which it cannot finish expressing is made to accompany the 4th as 
additional information by it as the above-mentioned application. 



When characteristic quantity has two or more values (ingredient) or information 
content is divided into two or more portions, One information unit can be expressed 
by two or more icons by admitting more than one as a feature vector, preparing 
another icon to each feature vector, or copying the same icon. 

The axiation part 40 is a portion which sets up a classification configuration space 
axis, sets up assignment on classification configuration space each axis of the 
characteristic quantity extracted from each media information, and sets up the 
classification configuration space provided with one or more axes. For example, three 
axes of the X-axis Y-axis Z-axis are specified, and let space which this XYZ axis 
stretches be classification configuration space. Here, it is displayed, for example so 
that the display display face of the information display sections 60 mentioned later 
may be an XY plane and screen depth may be Z shaft orientations. 
Here, the characteristic quantity which can be set as a classification configuration 
space axis should just be the characteristic quantity which can be extracted according 
to media, such as picture information, text information, and speech information. 
For example, about text information, the characteristic quantity which shows 
existence of a specific word, the characteristic quantity of the use count of a specific 
word, etc. can be mentioned. By setting the characteristic quantity which shows 
existence of a specific word, and the characteristic quantity of the use count of a 
specific word as a classification configuration space axis, the classification of 
contained-text information and arrangement of explanation of a specific word, etc. are 
attained. If it is the information set in which picture information is related with these 
text information, the picture information expressed by the specific word will also be 
classified and arranged collectively in classification configuration space. 
For example, about picture information, DCT transformation coefficient characteristic 
quantity, wavelet conversion factor characteristic quantity, HSI color histogram 
characteristic quantity, etc. are mentioned. By setting DCT transformation coefficient 
characteristic quantity as a classification configuration space axis, the classification of 
the information by the feature of the spatial frequency component of picture 
information and arrangement are attained. By setting wavelet conversion factor 
characteristic quantity as a classification configuration space axis, the classification of 
the information by the feature of the low frequency portion of spatial frequency, i.e., 



feature of the outside of the rereeling reel of the object especially reflected in a 
picture, of picture information and arrangement are attained. Although wavelet 
transforms are also a waveform and frequency conversion as well as DCT 
transformation, it can change with position (time) information maintained. By setting 
HSI color histogram characteristic quantity as a classification configuration space 
axis, the classification of the information based on the sexual desire news of a picture 
and arrangement are attained. According to the HSI color histogram, the feature of 
how many human being's skin fields are included and a picture can be held well. 
For example, about speech information, voice frequency characteristic quantity, audio 
amplitude characteristic quantity, audio time change characteristic quantity, etc. can 
be mentioned. By setting voice frequency characteristic quantity as a classification 
configuration space axis, the classification of speech information and arrangement are 
attained according to the feature of voice frequency, i.e., audio height, tone quality, 
etc. According to voice frequency, the exception of the cry of an animal, male voice, 
and female voice can express [ exception of a sounding body ] the feature of speech 
information, such as a person of voice, if accuracy goes up. By setting audio 
amplitude characteristic quantity and time change characteristic quantity as a 
classification configuration space axis, the classification of the speech information 
according to audio size and arrangement are attained. 

The assignment of characteristic quantity to the space axis of 1 by the axiation part 40 
can be assigned combining two or more characteristic quantity. Although the 
combination of two or more characteristic quantity needs to change and adjust the unit 
of mutual characteristic quantity, it transposes to mark like a score and it can be added 
together. For example, " 1 " and when it is not contained, to one space axis, if the color 
set ingredient of the HSI color histogram is contained at a rate beyond a threshold as 
the 1st characteristic quantity, it will be referred to as "0", and as the 2nd 
characteristic quantity, Processing of being referred to as "0" when it does not exist, 
"1" and if it exists as characteristic quantity which shows existence of the specific 
word in text information, adding together the 1st characteristic quantity and 2nd 
characteristic quantity, and assigning one space axis is possible. On the contrary, one 
characteristic quantity can also be assigned to two or more axes. Although many 
methods of assigning the axis of two or more one characteristic quantity [ from ] exist, 
the method (principal-component-analysis method) of considering that characteristic 



quantity is a vector, and distribution making multiple selection of the large dimension 
about each dimension of a vector, and centering upon the dimension as one of such 
the methods, is mentioned. 

The classification placement part 50 is a portion which carries out classification 
arrangement of the information set icon into classification configuration space based 
on the characteristic quantity extracted by the feature quantity extracting part 20. For 
example, if classification configuration space is set up with XYZ3 axis, an 
information set icon will be classified and arranged in the three-dimensional 
classification configuration space concerned. 

The information display sections 60 are portions which display the information set 
icon to the information set in which classification arrangement was carried out by the 
classification placement part 50 into classification configuration space. 
An information set icon is displayed on the position according to the viewpoint over 
classification configuration space. 

For example, if an XY plane is a transverse plane and makes Z shaft orientations a 
depth direction to the classification configuration space stretched from the XYZ axis, 
it will be displayed so that a display face may be an XY plane and screen depth may 
be Z shaft orientations. 

The example of the flow of processing of the multimedia information classification 
arranging apparatus of Embodiment 1 of this invention constituted as mentioned 
above is explained referring to drawing 4 . 

As shown in drawing 4 , divide roughly processing of the multimedia information 
classification arranging apparatus of Embodiment 1 of this invention, and as follows, 
The acquisition processing (SI 01) of an information set, and the generation 
processing (SI 02) of the information set icon corresponding to an information set, 
The axiation processing (SI 03) which sets up the characteristic quantity assigned to 
the axis of classification configuration space, and defines classification configuration 
space, The characteristic quantity extracting processing (SI 04) which extracts 
predetermined characteristic quantity from the media information of an information 



set, The information classification layout process (SI 05) which follows the extracted 
characteristic quantity, and classifies and arranges an information set icon in 
classification configuration space, The information-display processing (SI 06) which 
displays the information set icon classified and arranged from a setting-out viewpoint 
on classification configuration space and classification configuration space, The 
displayed search results were seen, and in order to continue search which changed the 
further narrowing down and cut end if needed, the characteristic quantity assigned to 
the axis of classification configuration space was reset, and it has the re-axiation 
processing (loop to Step SI 03) which redefines classification configuration space. 
When desired search results are obtained by the classification of an information set 
icon based on the classification configuration space set up by the first axiation 
processing (SI 03), and arrangement, re-axiation processing is not indispensable. 
Although premised on the information set existing in the accessible state from the 
beginning in the flow of the above-mentioned processing, The information in which 
the multimedia information classification arranging apparatus had relevance mutually 
among media information including picture information, text information, and speech 
information may be summarized so that it may mention later in Embodiment 2, and 
information set generation processing which defines and generates an information set 
may be performed as pretreatment. 

First, the multimedia information classification arranging apparatus of this invention 
performs information set acquisition processing (SI 01) by the information set 
acquisition part 10 with the retrieval execution by a user, etc. Here, by the information 
set acquisition part 10, information sets are collected from the sources 80 of 
multimedia information, such as a WWW server which is in a state accessible on the 
networks 70, such as the Internet, via a communication interface, and an information 
set is acquired, for example. In this example, two or more information sets in which 
picture information and text information are associated are acquired as information set 
acquisition processing (S101). In order to narrow down the range of the information 
set to acquire to some extent, only what was hit by retrieval by keyword may be 
acquired. 

Suppose that the information set containing the information set 3 hit in the keyword a 
"musician" was acquired in this example. For example, six information sets in which 



all contain sound data, image data, and text data, "-- a shoe belt ~ suppose that a 
peer's quintet music measure", the "Mozart piano concerto No. 20th", "marriage of 
Mozart Le Figaro", "rudder store E major with the sufficient Handel tune", "the Bach 
fantasy and a FUGATO minor key", "a BAHHA toccata, a FUGANI minor key", etc. 
were obtained. 

Next, a multimedia information classification arranging apparatus generates the 
information set icon corresponding to each information set by the information 
visualization part 30 (SI 02). The procedure is as having been shown in drawing 2 . 
Here, the icon which shows sound data to a compressed image and its upper part, and 
the icon which shows the contents of a text to the bottom are attached and 
compounded. The feature quantity extracting part 20 may be used in execution of the 
characteristic quantity extracting processing of information set icon generation. 
Next, a multimedia information classification arranging apparatus receives the input 
of the directions about information classification arrangement from a user, and starts a 
classification layout process. 

First, the characteristic quantity assigned to the axis of classification configuration 
space by the axiation part 40 is set up, and axiation processing (SI 03) which defines 
classification configuration space is performed. Suppose that characteristic quantity is 
not set to DCT transformation coefficient characteristic quantity [ as opposed to / as 
opposed to / to the X-axis (horizontal) / a tone / picture information to a Y-axis 
(perpendicular direction) ], and the Z-axis (depth direction) in this example. 
Next, a multimedia information classification arranging apparatus performs 
characteristic quantity extracting processing (SI 04) which extracts the characteristic 
quantity currently assigned to each space axis from each media information of the 
information set currently collected using the feature quantity extracting part 20. The 
feature quantity extracting part 20 extracts the characteristic quantity specified by the 
axiation processing SI 03 from each information set as mentioned above. In this 
example, the characteristic quantity of the tone of sound data and the DCT 
transformation coefficient characteristic quantity of image data are extracted. 
When characteristic quantity has two or more values (ingredient), two or more feature 
vectors can also be outputted. 



Next, a multimedia information classification arranging apparatus performs the 
information classification layout process (SI 05) which follows the extracted 
characteristic quantity using the classification placement part 50, and classifies and 
arranges an information set icon in classification configuration space. And 
information-display processing (SI 06) which displays the information set icon 
classified and arranged from a setting-out viewpoint on classification configuration 
space and classification configuration space by the information display sections 60 is 
performed. The example which has classified and arranged six information set icons 
collected by the keyword of the "musician" in classification configuration space is 
shown in drawing 5 . In this example, as for a piano, classification arrangement near a 
center and of the harp is carried out [ the X-axis ] for X-axis left-hand side and a 
violin near right-hand side as characteristic quantity of a tone, and a Y-axis 
presupposes that classification arrangement was carried out in the size of the DCT 
transformation coefficient characteristic quantity of image data. The information set 
icon of the example of drawing 5 is embellished by condition that an icon called a 
piano keyboard, a violin, and a harp is assigned according to the tone, or the icon 
portion of sound data is colored according to a minor key, a major key, etc. if sound 
data is taken for an example. 

When two or more feature vectors are outputted to one information set, One 
information unit is expressed by two or more icons by displaying a separate 
information set icon on the classification locating position to each feature vector, or 
copying the same information set icon. 

Desired picture information etc. are obtained from the above information set 
acquisition processing step SI 01 by the information-display processing step SI 06, 
The purpose of the original retrieval processing is attained, and when re-axiation 
processing is not required, (Step S107:NO), Although what is necessary is just to end 
retrieval processing, the purpose of the original retrieval processing is not attained, 
but when re-axiation processing is required, since re-axiation processing is performed, 
(Step SI 07: YES) and the multimedia information classification arranging apparatus 
of this invention can carry out a loop to Step SI 03. That is, in order to continue search 
which looked at classification / arrangement indication results displayed by the 



information-display processing step SI 06, and changed the further narrowing down 
and cut end if needed, The characteristic quantity assigned to the axis of classification 
configuration space is reset using the axiation part 40, and re-axiation processing 
which redefines classification configuration space is performed. Thus, the 
characteristic quantity assigned to a classification configuration space axis until 
required picture information is obtained is reset (SI 03), The characteristic quantity 
reset from each media information of the information set is re-extracted (SI 04), Based 
on the re-extracted characteristic quantity, redisplay of reclassification and the 
information set icon which rearranged (SI 05) and was rearranged by classification 
configuration space by the information display sections 60 is carried out for an 
information set icon to classification configuration space (SI 06). For example, it is 
also possible to change the X-axis to wavelet conversion factor characteristic quantity 
from the characteristic quantity of a tone, or to change a Y-axis to HSI color 
histogram characteristic quantity from DCT transformation coefficient characteristic 
quantity by the axiation part 40. Such a series of axial resetting processings are 
repeatable until desired search results are obtained. 

Thus, the indication results of a classification and arrangement of the information set 
icon obtained with a certain characteristic quantity can be seen, a classification and 
arrangement of an information set icon can be tried using the characteristic quantity of 
a completely different cut end, and a possibility that the heuristic new information 
which is not expected for a retrieving person will be acquired arises. 
It is also possible to see the indication results which are a classification and 
arrangement of information set icons, such as picture information, and to display the 
page of the web of the information set corresponding to the selected information set 
icon like drawing 6 . If URL in which information sets, such as the picture information 
concerned, are contained is saved collectively at a part of information set icon, when 
the picture information concerned is chosen from a user on the information display 
sections 60, it will become possible to display the web page based on URL 
information. For example, picture information is chosen and the web page is 
displayed by clicking from a menu etc. the button a "web page", with a pointing 
device. It is good also as displaying that the web page clicks the picture itself. 
Next, the flow of another different processing from the flow of the processing shown 
in above-mentioned drawing 4 is explained using drawing 7 . In the processing shown 



in this drawing 7 , the whole processing An information set icon generation phase, It 
divides into two phases of an information set search phase, and performs as 
pretreatment by making generation processing of an information set icon into an 
information set icon generation phase, and the information set icon to the information 
set used in an information set search phase is generated collectively. As an 
information set search phase, the search acquisition processing (701) of an 
information set, and axiation processing (S702), Characteristic quantity extracting 
processing (S703), an information classification layout process (S704), information- 
display processing (S705), and re-axiation processing (loop to Step S702) performed 
if needed are performed. If it is a flow of this processing, in an information set search 
phase, it is not necessary to generate an information set icon each time from the 
information set obtained as a result of search. It is only carrying out classification 
arrangement of the information set icon corresponding to the searched information set 
in classification configuration space, and since it is good, processing is accelerable. 
It performs in the information set generation phase by considering characteristic 
quantity extracting processing 703 as pretreatment, and to an information set, an 
information set icon and each characteristic quantity may be extracted, and may be 
matched and memorized. According to this composition, in an information set search 
phase, since it is not necessary to extract characteristic quantity from the information 
set obtained as a result of search each time, processing is further accelerable. 
As mentioned above, according to the multimedia information classification arranging 
apparatus shown in this Embodiment 1. By using the icon which compression 
efficiency is high, and there is little data transfer quantity as data used for presenting 
of information, and ends, The data volume to transmit can be reduced, and the waiting 
time to a retrieving person's information display can be shortened, and the list nature 
of information content to a retrieving person and grasp ease can be secured. A user's 
retrieval effectiveness can be raised by arranging to the neighborhood the information 
set icon related in an information set icon by carrying out classification arrangement 
in a feature space according to the characteristic quantity, or arranging a picture with 
the specific feature at a specific place. 

(Embodiment 2) 

Next, the multimedia information classification arranging apparatus of Embodiment 



2, Compound the icon corresponding to each media information which constitutes an 
information set like Embodiment 1, and an information set icon is not created, The 
icon for every media information is prepared, and it displays, changing an icon in time 
according to a display order in a locating position applicable in classification 
arrangement of an information set for every media information. 
The example of an equipment configuration of the multimedia information 
classification arranging apparatus of this Embodiment 2 is the same as that of drawing 

3. However, the contents of processing of a component differ. 
The information visualization part 30 does not carry out compounding the icon to 
these each media information, and generating an information set icon, although the 
icon which displays the contents visually is assigned to each media information which 
constitutes an information set. 

The classification placement part 50 carries out classification arrangement of the icon 
to each media information included in an information set in classification 
configuration space based on the attribute and the ingredient of characteristic quantity 
of the characteristic quantity of the media information which each information set has. 
An information set icon is not arranged but classification arrangement is carried out 
by making into a group the icon corresponding to each media information included in 
an information set. 

The information display sections 60 display classification configuration space and an 
icon, and they display them on a locating position, they determining the display order 
of the icon to each media information included in an information set, and changing 
these icons in time. For example, an icon is repeatedly changed and displayed every 
other second in order called the icon of image data, the icon of sound data, and the 
icon of text information. 

About the information set acquisition part 10, the feature quantity extracting part 20, 
the axiation part 40, the network 70, and the source 80 of multimedia information, it 
may be the same as that of what was explained by Embodiment 1. 
The example of the flow of processing of the multimedia information classification 
arranging apparatus of above Embodiment 2 is explained referring to drawing 8 . 
As shown in drawing 8 , processing of the multimedia information classification 



arranging apparatus of Embodiment 2 of this invention, The acquisition processing 
(S801) of an information set, and the generation processing (S802) of the icon of each 
media information, Axiation processing (S803) of classification configuration space, 
and the characteristic quantity extracting processing (S804) of an information set, It 
has a classification into the classification configuration space of the icon of each 
media information, a layout process (S805), classification configuration space and 
change display processing (S806) of the icon of each media information, and re- 
axiation processing (loop to Step S803). When desired search results are obtained by 
the classification and arrangement based on the classification configuration space set 
up by the first axiation processing (S803), re-axiation processing (S806) is not 
indispensable. 

A multimedia information classification arranging apparatus generates the icon 
corresponding to each media information included in each information set stored in 
the source 80 of multimedia information by the information visualization part 30 after 
performing acquisition processing (S801) of an information set (S802). Although the 
procedure is the same as that of what was shown in drawing 2 , it does not carry out 
compounding in the form of an information set icon eventually. The feature quantity 
extracting part 20 may be used in execution of the characteristic quantity extracting 
processing in icon generation of each media information. 

Axiation processing (S803) of classification configuration space and the characteristic 
quantity extracting processing (S804) of an information set are the same as that of 
S103-S104 of drawing 4 of Embodiment 1, and omit explanation. 
Next, a multimedia information classification arranging apparatus follows the 
extracted characteristic quantity using the classification placement part 50, and 
classifies and arranges the icon of each media information in classification 
configuration space (S805). And by the information display sections 60, changing 
classification configuration space and the icon of each media information, it classifies 
and arranges to classification configuration space, and displays on it (S806). 
When two or more feature vectors are outputted to one information set, one 
information unit is expressed by two or more icons by displaying an icon on the 
classification locating position to each feature vector. 



Desired picture information etc. are obtained from the above information set 
acquisition processing step S801 by the information-display processing step S806, 
The purpose of the original retrieval processing is attained, and when re-axiation 
processing is not required, (Step S807:NO), Although what is necessary is just to end 
retrieval processing, the purpose of the original retrieval processing is not attained, 
but when re-axiation processing is required, since re-axiation processing is performed, 
(Step S807:YES) and the multimedia information classification arranging apparatus 
of this invention can carry out a loop to Step S803. 

The whole processing like drawing 7 of Embodiment 1 An icon generation phase, It 
may divide into two phases of an information set search phase, and may perform as 
pretreatment by making generation processing of an icon into an icon generation 
phase, and the icon of each media information used in an information set search phase 
may be generated collectively. 

As mentioned above, according to the multimedia information classification arranging 
apparatus shown in this Embodiment 2. By using the icon which compression 
efficiency is high, and there is little data transfer quantity as data used for presenting 
of information, and ends, Since it displays being able to reduce the data volume to 
transmit, and being able to shorten the waiting time to a retrieving person's 
information display, and changing each icon according to a display order, the list 
nature of information content to a retrieving person and grasp ease are securable. 

(Embodiment 3) 

The multimedia information classification arranging apparatus of Embodiment 3 of 
this invention is explained. In the classification to the classification configuration 
space based on the characteristic quantity of the media information by a classification 
placement part, and arrangement, the multimedia information classification arranging 
apparatus of this Embodiment 3 applies a self-organization map. 

The example of an equipment configuration of this embodiment 3 multimedia- 
information classification arranging apparatus is shown in drawing 9 . As shown in 



drawing 9 , the classification placement part 50 is provided with the self-organization 
map treating part 51. About the information set acquisition part [ other than 
classification placement part 50 ] 10, information visualization part 30, axiation part 
40, feature quantity extracting part 20, and information-display-sections 60 network 
70, and the source 80 of multimedia information. It may be the same as that of the 
thing of the example of multimedia information classification arranging apparatus 
composition of drawing 3 explained by Embodiment 1 . 

A self-organization map is the teacher-less learning model which used the neural 
network. In a self-organization map, it maps from the feature vector space of high 
order origin to low dimension space. That to which the feature vector is similar in that 
case is arranged [ in low dimension space ] in the neighborhood. This self- 
organization map is applied to a classification and layout process of media 
information, self-organization map processing is used based on the characteristic 
quantity which the feature quantity extracting part 20 extracted, and the icon of each 
media information is arranged like an information set icon or Embodiment 2 like 
Embodiment 1. The self-organization map treating part 51 performs the self- 
organization map generation processing concerned to the characteristic quantity 
which the feature quantity extracting part 20 extracted. The classification placement 
part 50 of this Embodiment 3 is classified and arranged in classification configuration 
space based on the arrangement information acquired on the self-organization map 
which the self-organization map treating part 5 1 generated. It can combine with self- 
organization map processing, and the depth expression based on the characteristic 
quantity currently assigned to the Z-axis (depth direction) can also be combined. For 
example, based on text information, it decomposes into the frequency information of a 
word, each frequency is vectorized, and the position of the axis of a depth direction is 
determined based on the vector. When Web is searched using a keyword as other 
examples, based on the degree of association of the keyword which a text browsing 
server returns, and a Web page, it can also display on order with a high degree of 
association as depth information, and they can be changed and displayed. 
As mentioned above, according to the multimedia information classification arranging 
apparatus of this Embodiment 3, self-organization map processing can be applied and 
what what is considered for the contents to be near on space is considered for the 



contents to be far by by the neighborhood can arrange a picture in the distance, 
respectively. 

(Embodiment 4) 

The multimedia information classification arranging apparatus of Embodiment 4 of 
this invention is explained. The multimedia information classification arranging 
apparatus of this Embodiment 4 adds a device to the method of presentation of the 
classification configuration space where the information set icon has been classified 
and arranged, and a browsing method. 

The example of an equipment configuration of the multimedia information 
classification arranging apparatus of this Embodiment 4 is shown in drawing 10 . As 
shown in drawing 10 , the information display sections 60 are provided with the 
display view point movement part 61. About information set acquisition parts 10 
other than information-display-sections 60, the information visualization part 30, the 
axiation part 40, the feature quantity extracting part 20, the classification placement 
part 50, the network 70, and the source 80 of multimedia information. It may be the 
same as that of the thing of the example of multimedia information classification 
arranging apparatus composition of drawing 3 explained by Embodiment 1 . 

In displaying the classification configuration space where the information set icon has 
been classified and arranged by the classification placement part 50, the display view 
point movement part 61 is provided with the function to which the setting-out position 
of the viewpoint which displays the space concerned is moved in the information 
display sections 60. The information display sections 60 display the classification 
configuration space seen from the display viewpoint set up by the display view point 
movement part 61 . 

In the multimedia information classification arranging apparatus of this Embodiment 
4, the display viewpoint of drawing 5 shown by Embodiment 1 is considered as a 
default, and it becomes possible to change a display viewpoint dynamically by the 
display view point movement part 61. So to speak, an information set icon can move 
about the inside of the classification configuration space classified and arranged freely 



virtually, and a classification and signs that it has been arranged of the information set 
icon in the classification configuration space which is visible from a movement zone 
can be displayed. Since a display display surface is a two-dimensional flat surface 
fundamentally if it is usual, although the display in perspective is possible, it tends to 
be hard to see the information set icon in a position with deep depth. However, 
according to the multimedia information classification arranging apparatus of this 
Embodiment 4, a display viewpoint can be changed dynamically and a retrieving 
person can display the situation of a classification and arrangement of an information 
set icon to see in more detail near the display display surface according to a retrieving 
person's operation. 

(Embodiment 5) 

The multimedia information classification arranging apparatus of Embodiment 5, The 
classification configuration space which resets assignment of the attribute of the 
characteristic quantity to classification configuration space each axis, and has already 
been displayed is reset, In displaying signs that carry out reclassification arrangement 
of each information set icon to the reset classification configuration space, and 
reclassification arrangement of each information set icon is carried out, from the 
position currently displayed before reclassification arrangement to the position 
displayed after reclassification arrangement, It is made to move by predetermined unit 
and the media information currently displayed is displayed. 

The function which fixes to the specific position on the classification configuration 
space specified by a user the information set icon which the user selected in 
reclassification arrangement, When carrying out reclassification arrangement of each 
information set icon to the classification configuration space where the axis was reset, 
the information set icon which the user selected is fixed to a specific position, and it 
has a function which carries out reclassification arrangement only of other 
information set icons according to classification configuration space. 

Drawing 1 1 shows briefly the equipment configuration of the multimedia information 
classification arranging apparatus concerning Embodiment 5. As shown in drawing 
11, the information display sections 60 are provided with the animation treating part 



62. the classification placement part 50 — a locating position fixed digit — a law ~ it 
has the part 52. About information set acquisition parts 10 other than classification 
placement part 50 and information-display-sections 60, the information visualization 
part 30, the axiation part 40, the feature quantity extracting part 20, the network 70, 
and the source 80 of multimedia information. It may be the same as that of the thing 
of the example of multimedia information classification arranging apparatus 
composition of drawing 3 explained by Embodiment 1. 

In the animation treating part 62 of the information display sections 60 displaying 
signs that reclassification arrangement of each information set icon is carried out on 
classification configuration space by the re set of an axis, It has a function which 
makes it move by predetermined unit and displays each information set icon to the 
position displayed after reclassification arrangement from the position currently 
displayed before reclassification arrangement. For example, the position before 
reclassification arrangement of each information set icon is memorized, A notice is 
received for the position after reclassification arrangement of each information set 
icon from the classification placement part 50, and the move direction and migration 
length are found for every information set icon based on both coordinate points, and 
each information set icon is moved in the unit of predetermined, and it goes. A 
predetermined unit is good irrespective of the number of movable steps also as a unit 
which adjusted migration length so that movement might be completed by the 
predetermined number of steps well also as movement of predetermined every 
distance. 

the classification placement part 50 ~ a locating position fixed digit — a law — having 
the part 52 — a user — a locating position fixed digit — a law — a specific information 
set icon can be specified that it fixes to a specific position via the part 52. When the 
classification placement part 50 carries out reclassification arrangement of each 
information set icon to the classification configuration space where the axis was reset, 
it is in the state which fixed the specified information set icon to the specific position, 
and is provided with the function which carries out reclassification arrangement only 
of other information set icons according to classification configuration space. 
The above-mentioned reclassification arrangement function which moves each 
information set icon and goes by predetermined unit, and the reclassification 



arrangement function which fixed the specific information set icon to the specific 
position are explained using the example of drawing 5 and drawing 12 . 
Drawing 5 explained by Embodiment 1 is used as an example of the displaying 
condition of the information set icon in the classification configuration space before 
reclassification arrangement. Suppose that characteristic quantity is not set to DCT 
transformation coefficient characteristic quantity [ as opposed to / as opposed to / to 
the X-axis (horizontal) / a tone / picture information to a Y-axis (perpendicular 
direction) ], and the Z-axis (depth direction) in this example. 

Next, signs that reclassification arrangement is carried out are explained. HSI color 
histogram characteristic quantity should be assigned by re-axiation as an axis of the 
direction of an XY plane, and "0" values should be altogether assigned to especially 
the Z-axis, without assigning characteristic quantity. It is drawing 12 which showed 
the display example after the reclassification arrangement displayed by the 
information display sections 60. 

Thus, along with the characteristic quantity newly assigned to the axis, 
reclassification arrangement of the information set icon is carried out by re-axiation. 
Here, to the position displayed after reclassification arrangement from the position 
currently displayed before reclassification arrangement, the animation treating part 62 
is moved by predetermined unit, and displays each information set icon. That is, from 
the state of drawing 5 to the state of drawing 12 . predetermined minces each 
information set icon, it is displayed from the position before reclassification 
arrangement to the position after reclassification arrangement, and signs that it moves 
so that an information set icon can follow by the eye are displayed. The animation 
treating part 62 memorizes the position before reclassification arrangement of each 
information set icon shown in drawing 5 . The notice of the position after 
reclassification arrangement of each information set icon shown in drawing 12 is 
received, and the move direction and migration length are found for every information 
set icon based on both the coordinates position, and the media information as which 
each information set icon is displayed in the unit of predetermined is moved, and it 
goes. 



Next, the locating position fixed digit of the classification placement part 50 — a law - 

- the example of reclassification arrangement in the state where the specific 
information set icon using the part 52 was fixed to the specific position is explained. 
A user selects one or more information set icons from the inside of the screen 
currently displayed, in order to grasp the relation between a specific information set 
and other information sets via the locating position fixed specification part 52. For 
example, one information set icon is selected, the information set icon is fixed in the 
center, and other information set icons are arranged in classification configuration 
space by the self-organization map method. Select two or more information set icons, 
for example, four information set icons, and the physical relationship of these 
information set icon is fixed to specific physical relationship, for example, the four 
corners of a screen, It is also possible to arrange other information set icons in 
classification configuration space by the self-organization map method. Thus, by 
fixing a specific information set icon to specific physical relationship, and performing 
classification arrangement, what kind of relation the selected information set and each 
information sets of other have can grasp now easily visually. For example, if a red 
picture, a blue picture, a yellow picture, and a green picture are fixed to four corners, 
respectively and classification arrangement is carried out when the axis of 
classification configuration space makes sexual desire news characteristic quantity 
now, The pictures of a red system gather for the corner side with the fixed red picture, 
and since a purple picture is gathered and displayed near the middle of the position of 
a red picture, and the position of a blue picture, it becomes easy to find each picture 
along with the system of a color. 

Drawing 13 and drawing 14 are the figures showing the concept of the reclassification 
arrangement which fixed to the specific position the specific information set icon 
which used the locating position fixed specification part 52. setting drawing 13 before 
reclassification arrangement — a locating position fixed digit — a law — the figure 
showing signs that selected designation of the specific information set icon was 
carried out using the part 52. It is a figure showing signs that drawing 14 fixed to the 
specific position (middle of the screen) the information set icon by which selected 
designation was carried out by drawing 13 , and carried out reclassification 
arrangement of other information set icons, in drawing 13 — the information set icon 5 

— a user — a locating position fixed digit — a law — it is the information set icon by 



which selected designation was carried out via the part 52. This information set icon 5 
is fixed in the center, and if reclassification arrangement is carried out with a different 
attribute from reclassification before, for example, sexual desire news, the 
information set icons which have the tone near the information set icon 5 like drawing 
14 will gather near a center. 

The reclassification arrangement function which fixed the specific information set 
icon using this locating position fixed specification part 52 to the specific position is 
received, It is also possible to combine the reclassification arrangement function 
which moves each information set icon in the unit of predetermined, and goes, It is 
possible to also make it display so that each information set icon may move in the unit 
of predetermined to the position after reclassification in the case of the reclassification 
arrangement which fixed the specific information set icon to the specific position. 
According to the above-mentioned composition, classification configuration space 
can be reset and reclassification of the information set icon to observe can be carried 
out in the state where it was fixed to the specific position, The situation how the 
locating position of each information set icon changed can be checked visually, and 
the convenience of search of an information set icon and sorting processing improves. 

(Embodiment 6) 

The multimedia information classification arranging apparatus of Embodiment 6 of 
this invention is explained. As for the multimedia information classification arranging 
apparatus of this Embodiment 6, the multimedia information classification arranging 
apparatus of Embodiments 1-5 mentioned above is built by the client server system 
through a computer network. Some patterns are possible in which element which 
element is included in a server computer among the equipment configurations of the 
multimedia information classification arranging apparatus of Embodiments 1-5 
mentioned above, and is included in a client computer. 
Drawing 1 5 is the example of client-server composition which formed the information 
visualization part 30, the information set acquisition part 10, and the feature quantity 
extracting part 20 in the server computer 100, and formed the axiation part 40, the 
classification placement part 50, and the information display sections 60 in the client 
computer 101. 



Drawing 16 is the composition of having formed the information visualization part 
30, the information set acquisition part 10, the feature quantity extracting part 20, and 
the classification placement part 50 in the server computer 100, and having formed 
the axiation part 40 and the information display sections 60 in the client computer 
101. 

However, in above-mentioned drawing 15 - each composition of drawing 16 , several 
kinds of methods are mentioned about the acquisition method of an information set. 
For example, there are automatic collection and the method of updating automatically 
periodically from on a network etc. about the information set of the contents in which 
the server computer 1 00 is beforehand specified as a robot. Otherwise, although the 
information set acquisition part 10 is illustrated by the server computer 100, each of 
drawing 15 - drawing 16 , Separate the information set collection key input section 14 
from the information set collecting part 12 of the information set acquisition part 10, 
and it provides in the client computer 101 (not shown), The user of the client 
computer 101 inputs an information set collection key using the information set 
collection key input section 14, The inputted information set collection key concerned 
is passed to the information set acquisition part 10 of the server computer 100, and 
there is a method of collecting dynamically the information sets in which the server 
computer 100 corresponds using the information set collection key concerned from on 
a network etc. 

Thus, the multimedia information classification arranging apparatus of this invention 
can be built by a client/server system by dividing into a server computer and 
KURAAN ipecac pewter the equipment configuration element of the multimedia 
information classification arranging apparatus of Embodiments 1-5 mentioned above, 
and establishing it. 

(Embodiment 7) 

The multimedia information classification arranging apparatus of this invention can 
be built using various computers by recording the program which described the 
processing step which realizes processing explained to the above-mentioned 



embodiment on the recording medium in which computer reading is possible, and 
providing it. The recording medium which recorded the program provided with the 
processing step which realizes the multimedia information classification arranging 
apparatus of this invention, As shown in the example of the recording medium 
illustrated to drawing 17, not only in the portability type recording medium 201 of 
CD-ROM202 or flexible disk 203 grade, It may be any of the recording media 205, 
such as the recording medium 200 in the recorder on a network, a hard disk of a 
computer, and RAM, and at the time of program execution, loading of the program is 
carried out on the computer 204, and it is executed on main memory. 

Industrial applicability 

By using the information set icon which compression efficiency is high, and there is 
little data transfer quantity as data used for presenting of information, and ends 
according to the multimedia information classification arranging apparatus of this 
invention, The data volume to transmit can be reduced, and the waiting time to a 
retrieving person's information display can be shortened, and the list nature of 
information content to a retrieving person and grasp ease can be secured. A user's 
retrieval effectiveness can be raised by arranging to the neighborhood the information 
set icon related in an information set icon by carrying out classification arrangement 
in a feature space according to the characteristic quantity, or arranging a picture with 
the specific feature at a specific place. It can replace with an information set icon and 
the list nature of information content to a retrieving person and grasp ease can be 
secured also in the case where it displays changing the icon of each media information 
according to a display order. 

By holding an information set generation part according to the multimedia 
information classification arranging apparatus of this invention, In advance of a 
classification of an information set and a layout process, media information including 
picture information, text information, and speech information can be collected as 
pretreatment, the collected relevance between media information can be analyzed, 
related media information can be summarized, and an information set icon can be 
generated. 



According to the multimedia information classification arranging apparatus of this 
invention, classification configuration space is reset, Reclassification of the 
information set icon to observe can be carried out in the state where it was fixed to the 
specific position, the situation how the locating position of each information set icon 
changed can be checked visually, and the convenience of search of an information set 
and sorting processing improves. 

[Brief Description of the Drawings] 

The figure in which (a) of Fig. 1 showed briefly the concept of the "information set" 
of this invention, and (b) are the figures showing the example of the multimedia 
information in conventional technology. 

Fig. 2 is a figure showing the example of an information set icon, and an example of 
the generation procedure of the information set icon concerned. 

Fig. 3 is a figure showing the example of an equipment configuration of the 
multimedia information classification arranging apparatus of Embodiment 1 of this 
invention. 

Fig. 4 is a flow chart which shows the example of the flow of processing of the 
multimedia information classification arranging apparatus of Embodiment 1 of this 
invention. 

Fig. 5 is a figure showing the example which classified six information set icons 
collected by the keyword of the "musician" in classification configuration space, and 
has been arranged. 

Fig. 6 is a figure showing the page of the web of the information set corresponding to 
the selected information set icon. 

The flow chart with which Fig. 7 shows the example of the flow of processing of the 
multimedia information classification arranging apparatus of Embodiment 2 of this 
invention 

Fig. 8 is a figure showing the example of an equipment configuration of the 
embodiment 3 multimedia-information classification arranging apparatus of this 
invention. 

Fig. 9 is a figure showing the example of an equipment configuration of the 
multimedia information classification arranging apparatus of Embodiment 4 of this 



invention. 

Fig. 10 is a figure showing briefly the equipment configuration of the multimedia 
information classification arranging apparatus concerning Embodiment 5 of this 
invention. 

Fig. 1 1 is a figure showing the display example after the reclassification arrangement 
displayed by the information display sections 60 concerning Embodiment 5 of this 
invention. 

setting Fig. 12 before reclassification arrangement — a locating position fixed digit — 
a law — it is a figure showing signs that selected designation of the specific 
information set icon was carried out using the part 52. 

Fig. 13 is a figure showing signs that fixed to the specific position (middle of the 
screen) the information set icon by which selected designation was carried out by Fig. 
12, and reclassification arrangement of other information set icons was carried out. 
Fig. 14 is a figure showing the example which built the multimedia information 
classification arranging apparatus of Embodiment 6 of this invention with client- 
server composition. 

Fig. 15 is a figure showing other examples which built the multimedia information 
classification arranging apparatus of Embodiment 6 of this invention with client- 
server composition. 

Fig. 16 is a figure showing the example of the recording medium which recorded the 
program which used the contents of processing of the multimedia information 
classification arranging apparatus of Embodiment 7 of this invention as the processing 
step. 

(57)[Claim(s)] 
[Claim 1] 

Let what carried out grouping of congener and different-species media information 
which relate to the same object from a set of media information including picture 
information, text information, and speech information be an information set, 
An information set acquisition part which acquires media information by making said 
information set into a unit, 

A feature quantity extracting part which extracts an ingredient of characteristic 
quantity from each media information of each information set, 
A rule which changes into an icon characteristic quantity of data in which 



correspondence with characteristic quantity of media information and an icon is 
shown, or media information is used . An information visualization part which 
generates an information set icon which assigns an icon which displays the contents 
visually to each media information which constitutes said information set, compounds 
these icons, and expresses the contents of said information set visually, 
An axiation part which sets up classification configuration space which assigned an 
attribute chosen from the attribute of characteristic quantity extracted from each 
media information included in each acquired information set as an axis of space 
which arranges an information set icon, and was provided with one or more axes, 
A classification placement part which carries out classification arrangement of the 
information set icon into said classification configuration space based on an attribute 
and an ingredient of characteristic quantity of characteristic quantity of media 
information which each information set has, 

A multimedia information classification arranging apparatus provided with 
information display sections which display said classification configuration space and 
said information set icon. 

[Claim 2] 

Let what carried out grouping of congener and different-species media information 
which relate to the same object from a set of media information including picture 
information, text information, and speech information be an information set, 
An information set acquisition part which acquires media information by making said 
information set into a unit, 

A feature quantity extracting part which extracts an ingredient of characteristic 
quantity from each media information of each information set. 
An information visualization part which assigns an icon which displays the contents 
visually to each media information which constitutes said information set using a rule 
which changes into an icon characteristic quantity of data in which correspondence 
with characteristic quantity of media information and an icon is shown, or media 
information . 

An axiation part which sets up classification configuration space which assigned an 
attribute chosen from the attribute of characteristic quantity extracted from each 
media information included in each acquired information set as an axis of space 
which arranges an icon, and was provided with one or more axes, 



A classification placement part which carries out classification arrangement of the 
icon to each media information included in said information set in said classification 
configuration space based on an attribute and an ingredient of characteristic quantity 
of characteristic quantity of media information which each information set has, 
They are information display sections which display said classification configuration 
space and said icon, A multimedia information classification arranging apparatus 
provided with information display sections displayed on said locating position 
deciding a display order of an icon to each media information included in said 
information set, and changing these icons in time. 

[Claim 3] 

When two or more ingredients of characteristic quantity which said feature quantity 
extracting part extracted exist about media information with a certain information set, 
said classification placement part, The multimedia information classification 
arranging apparatus according to claim 1 which carries out classification arrangement 
of said information set icon at two or more positions based on two or more ingredients 
of each of said characteristic quantity, respectively. 

[Claim 4] 

When two or more ingredients of characteristic quantity which said feature quantity 
extracting part extracted exist about media information with a certain information set, 
said classification placement part, The multimedia information classification 
arranging apparatus according to claim 2 which carries out classification arrangement 
of the icon to media information included in said information set at two or more 
positions based on two or more ingredients of each of said characteristic quantity, 
respectively. 

[Claim 5] 

Said information display sections are provided with a display view point movement 
part with a function to which a setting-out position of a viewpoint which displays an 
information set and classification configuration space is moved, 
The multimedia information classification arranging apparatus according to claim 1 or 
2 with which an information set icon displays classification configuration space 



classified and arranged based on a view position where said display view point 
movement part; set up said information display sections. 

[Claim 6] 

The multimedia information classification arranging apparatus according to claim 1 or 
2 with which said information display sections display said classification 
configuration space and said information set icon on a terminal unit through a 
network. 

[Claim 7] 

The multimedia information classification arranging apparatus according to claim 1 or 
2 assigned by choosing either of the icons currently prepared beforehand in 
assignment of 'an icon by said information visualization part. 

[Claim 8] 

The multimedia information classification arranging apparatus according to claim 7 
which chooses either of the icons currently prepared beforehand and embellishes 
display information of said selected icon in assignment of an icon by said information 
visualization part according to the contents of corresponding media information. 

[Claim 9] 

A thing with which the attribute of characteristic quantity assigned to one space axis 
in said axiation part combined the attribute of two or more characteristic quantity, or 
the multimedia information classification arranging apparatus according to claim 1 or 
2 to which the attribute of one characteristic quantity is assigned to two or more axes. 

[Claim 10] 

Said axiation part resets assignment of the attribute of characteristic quantity to each 
axis of a classification configuration space axis, and is provided with an axial 
resetting function which resets classification configuration space provided with one or 
more axes, 
Said feature quantity extracting part extracts an ingredient of characteristic quantity 
according to an axial re set by said axiation part, The multimedia information 
classification arranging apparatus according to claim 1 or 2 with which said 



classification placement part carries out classification arrangement of each 
information set icon into said classification configuration space according to an 
ingredient of said extracted characteristic quantity, and said information display 
sections display media information according to a viewpoint over said reset 
classification configuration space. 

[Claim 11] 

Said classification placement part is provided with a self-organization map treating 
part which performs self-organization by study using a partial interaction, 
The multimedia information classification arranging apparatus according to claim 1 or 
2 with which said classification placement part arranges an information set icon based 
on characteristic quantity which said feature quantity extracting part extracted using 
self-organization map processing by said self-organization map treating part. 

[Claim 12] 

A function in which said classification placement part fixes to a specific position on 
classification configuration space specified by a user an information set icon which a 
user selected, 
When carrying out reclassification arrangement of each information set icon to 
classification configuration space where an axis was reset, The multimedia 
information classification arranging apparatus according to claim 1 provided with a 
function which fixes to said specific position an information set icon which said user 
selected, and carries out reclassification arrangement only of other information set 
icons according to classification configuration space. 

[Claim 13] 

It is the recording medium which recorded a processing program which realizes a 
multimedia information classification arranging apparatus which carries out 
classification arrangement into classification configuration space, and displays an 
information set which is a set of information which had relevance mutually among 
media information including picture information, text information, and speech 
information and in which computer reading is possible, 
Information set acquisition processing which uses as an information set what carried 
out grouping of congener and different-species media information which relate to the 



same object from a set of media information including picture information, text 
information, and speech information, and acquires media information by making said 
information set into a unit, 

Characteristic quantity extracting processing which extracts an ingredient of 
characteristic quantity from each media information of each information set 
A rule which changes into an icon characteristic quantity of data in which 
correspondence with characteristic quantity of media information and an icon is 
shown, or media information is used . An information imaging process which 
generates an information set icon which assigns an icon which displays the contents 
visually to each media information which constitutes said information set, compounds 
these icons, and expresses the contents of said information set visually, 
Axiation processing which sets up classification configuration space which assigned 
an attribute chosen from the attribute of characteristic quantity extracted from each 
media information included in each acquired information set as an axis of space 
which arranges an information set icon, and was provided with one or more axes, 
A classification layout process which carries out classification arrangement of the 
information set icon into said classification configuration space based on an attribute 
and an ingredient of characteristic quantity of characteristic quantity of media 
information which each information set has, 

A storage memorizing a processing program which makes a computer perform 
information-display processing which displays said classification configuration space 
and said information set icon. 



[Claim 14] 

It is a multimedia information classification configuration method which carries out 
classification arrangement into classification configuration space, and displays an 
information set which is the set of information whose computer had relevance 
mutually among media information including picture information, text information, 

and speech information . 

Said computer uses as an information set what carried out grouping of congener and 
different- species media information which relate to the same object from a set of 
media information including picture information, text information, and speech 
information, and acquires media information by making said information set into a 
unit, 



Said computer extracts an ingredient of characteristic quantity from each media 

information of each information set, 

A rule from which said computer changes into an icon characteristic quantity of data 
in which correspondence with characteristic quantity of media information and an 
icon is shown, or media information is used . An icon which displays the contents 
visually is assigned to each media information which constitutes said information set, 
and an information set icon which compounds these icons and expresses the contents 
of said information set visually is generated . 

An attribute which said computer chose from the attribute of characteristic quantity 
extracted from each media information included in each acquired information set is 
assigned as an axis of space which arranges an information set icon, and classification 
configuration space provided with one or more axes is set up, 
Said computer carries out classification arrangement of the information set icon into 
said classification configuration space based on an attribute and an ingredient of 
characteristic quantity of characteristic quantity of media information which each 
information set has, 

A multimedia information classification configuration method, wherein said computer 
displays said classification configuration space and said information set icon. 
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A. . - INFORMATION SET 3 

8... MOZART: PIANO CONCERTO No. 20 

C. . .A music CD— ROM that Mozart fans and music fans have been 
eagerly awaiting is now debuting. The beginning parts of 
all his works including even f-ragments are recorded for 
the first time in the world from ""the Complete Works of 
Mozart" recorded on 190 CDs - Here, Piano Concert No. 20, 
D minor, KV. ACS is being played as an example. 



(57) Abstract: An information sorting/arranging device contributing to reduction of the amount of data transferred, and ensuring 
the brief graspability of information contents at a glance and the grasp easiness when information is displayed on a client device. An 
information set acquiring unit (10) acquires an information set. An information visualizing 
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unit (30) creates an information set icon. The information set is a group of media information concerning the same object and 
selected from a variety of media information subsets. The information set icon is an icon visually representing the contents of the 
information set and created by combining icons allocated to the media information subsets and visually representing the contents of 
the media information subsets. An axis setting unit (40) allocates a feature to a sorting/arranging space axis. A feature extracting 
unit (20) extracts a feature from each media information subset. A sorting/arranging unit (50) arranges the information set icons in 
a sorting/arranging space according to the respective features. An information display unit (60) displays them. 
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